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REMARKS 

The Office Actions dated January 22, 2003 and May 9, 2002 have been carefully 
reviewed. In consideration of the new matter rejection in the May 9, 2002 Office Action, 
Applicant has resubmitted the original specification as a new substitute specification. For the 
Examiner's convenience, a clean copy of the substitute specification submitted with this 
Amendment, which is also the original specification filed on March 31, 1999, is also 
submitted with this Amendment. Claims 14-47 have also been cancelled. The new substitute 
specification and the cancellation of claims submitted with the November 12, 2002 make the 
new matter rejections in the May 9, 2002 Office Action moot. 

With the present Amendment, Applicant adds new claims 48-90. New claims 48-60 
correspond substantially to original claims 1-13. Therefore, Applicant respectfully requests 
reconsideration of the grounds of rejection in the May 9, 2002 Office Action in view of the 
above amendments and the remarks herein. 

In the earlier Office Action dated May 9, 2002, claims 1-3, 5-7, 10-1 1 and 13 were 
rejected under 35 U.S.C. § 1 02(b) as being anticipated by U.S. Pat. No. 4,841,387 to 
Rindfuss. Claims 4, 8-9 and 12 were rejected under 35 U.S.C. § 103(a) as being unpatentable 
over Rindfuss in view of U.S. Pat. No. 5,172,28 1 to Ardis et al. New claims 48-60 
substantially correspond to these original claims, respectively. 

The claims, as amended and re-presented, require an assessment system that is 
comprised of at least one predefined measurement item used to structure observation and 
assessment of observable events or phenomena. The system further comprises means for 
presenting a subject or operator with the predefined measurement item , and means for 
recording at least one observable event associated with die presented measurement item and 
storing the recorded observable event as a track of event data sequenced by a track of address 
data. For each presented measurement item, the system further comprises means for 
associating the measurement item to at least one portion of the event data in an index 
database, the index database including at least one field denoting the measurement item and 
at least one field denoting address data for the portion of the event data associated with the 
measurement item. 

Rindfuss teaches linking handwritten notes to a recording through indexes based upon 
the physical location of the handwriting on a page of paper to the stream of recorded data. 
The system in Rindfuss uses a touchpad located beneath the paper to acquire spatial location 
data, further indexes by page number. When amendments or additions to notes are made, 
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additional index entries are generated and maintained, requiring manual entry of the previous 
page number if a prior piece of paper is the target of the amendment or addition to the notes. 
Replay requires manual entry of page numbers and physical contact with the paper in the 
approximate area of the recorded location to generate a touchpad signal that can be used to 
access the correlated recording. 

The May 9, 2002 Office Action states that Rindfuss teaches presenting a subject or 
operator with a measurement item at Col. 1, line 59 through Col. 2, line 2. However, 
. Rindfuss only refers to events and handwritten notes. Neither events nor handwritten notes 
are equivalent to predefined measurement items, as required by the claims, as amended. A 
"measurement item", as cited in independent Claims 48, 54, 80 and 85, is "used to structure 
observation and assessment of at least one observable event or phenomena." Measurement 
items, as used in the present invention, are predefined terms and/or instructions used as a tool 
or instrument for the purpose of systematic and standardized observation and assessment. 
The handwritten notes in Rindfuss do not contain instructions for the purpose of systematic 
and standardized observation and assessment, and are certainly not predefined. As described 
in the present application at Page 10, lines 20-27, the measurement items presented by the 
inventive system may be a series of questions to be used as a guide for an assessment 
interview. In this example, each question is a "measurement item", and the present invention 
indexes the recorded event data track for each measurement item, i.e. question. Measurement 
items may also include definitions, inquiries, instructions, directions or other tasks. (Page 12, 
line 18). 

Significantly, measurement items used in the present invention are predefined In one 
embodiment, the predefined measurement items are pre-existing (Page 1 1, lines 1-4), and are 
also used independently of the inventive system (see also SCAN example, Page 1 1, lines 20- 
28). In an alternative embodiment, an interview designer generates a set of measurement 
items prior to the event or events to be recorded (Page 12, lines 17-29). In all cases, the 
measurement items are defined before use with the inventive method and system. The 
inventive method and system relies on the measurement items to structure the systematic and 
standardized observation and assessment of events or phenomena. 

Furthermore, the claimed invention requires associating predefined measurement 
items with a portion of the recorded events through an index database to form a digital record 
of indexed inquiries and indexed responses. (Page 11, lines 9-20). This index provides the 
means to compare events from different subjects or groups of subjects on the basis of the 
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associated measurement items in order to investigate potential relationship and/or trends, as 
well as the means to organize an assessor's notes. The ability of the inventive system to 
perform real-time indexing as events occur, as well as post-processing organization of 
information is not taught or suggested by Rindfuss or Ardis. 

Ardis likewise fails to teach or suggest predefined measurement items used to 
structure the observation and assessment of observable events or phenomena, wherein the 
record of at least one observable event associated with a measurement item is automatically 
indexed in a database linking the measurement item to a portion of the event record* The 
system in Ardis only discloses post-processing of transcripts, and is in no way related to the 
automatic generation of an index during recording. Ardis only teaches a separate software 
system for laborious manual entering of index information after a deposition has been 
recorded. 



Applicant respectfully requests favorable reconsideration of this application, as 
amended. Claims 48-90 are pending, with Claims 48, 54, 80 and 85 being independent 
claims. 

Should the Examiner believe that further amendments are necessary to place the 
application in condition for allowance, or if the Examiner believes that a personal interview 
would be advantageous in order to more expeditiously resolve any remaining issues, the 
Examiner is invited to contact Applicant's undersigned attorney at the telephone number 
listed below. 

To the extent necessary, Applicant petitions for an extension of time under 37 CFR § 
1.136. Please charge any shortage in fees due in connection with this application, including 
extension of time fees, to Deposit Account No. 50-1 165 (Attorney Docket No G2622- 
908496) and credit any excess fees to the same Deposit Account. 



Summary 



Respectfully submitted, 




MILES & STOCKBRIDGE P.C. 
1751 Pinnacle Drive, Suite 500 
McLean, VA 22102*3833 
(703) 903-9000 



Registration No. 46,642 
Agent for Applicant 



OCT 0 7 2003 
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SYSTEM AND METHOD FOR INDEXING TEST EVENTS 



This application is a continuation in part of provisional patent application serial no. 
60/127,194 on March 31, 1999, the disclosure of which is herein incorporated by reference. 

This invention was made with a grant from the U.S. government which may have rights 
to the invention. 

FIELD OF THE INVENTION 

The present invention generally relates to information processing systems. More 
particularly, the invention is a data capture and access system that correlates data with recorded 
signals with an index based on pre-identified identifiers at the time of capture but allows later 
annotation or additional indexing without affecting the original index identifiers or the recorded 
signals. The invention specifically provides a method for- indexing of digital recordings of 
events by use of pre-entered or pre-existing measurement items that serve as index information 
designators such as key words or labels. Such a method is particularly useful as those used in a 
systematic interview or assessment method. 

DESCRIPTION OF RELATED ART 

Many events in the medical, legal, business and education domains, among others, 
today are being digitally recorded. Physicians may record a physical examination, operation, or 
autopsy and need to annotate that recording at some later time to notes, observations, or other 
content in order to make a more complete record or draw previously unseen trends. Lawyers 
may seek to record a witness deposition and later link that oral testimony to evidence 
(documentary, video, or testimony by other witnesses) in order to draw parallels or highlight 
inconsistencies. The frequency of such recordings, audio, video, or other, is likely to increase as 
their usefulness continues to grow thereby allowing for review of the assessor's conclusion or 
diagnoses, providing an educational or teaching tool to illustrate optimum assessment interview 
techniques, documenting the interview for legal and/or other purposes. - 
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Medicine and public health are domains in which considerable effort has been devoted 
to the development of assessment schedules and instruments for conducting assessment 
interviews. The objective of these efforts has often been to improve the reliability and efficiency 
of a clinician's diagnosis of his/her patient 

Among medical disciplines, psychiatry researchers have perhaps allocated the most 
efforts to developing patient assessment aids. Examples of the types of instruments developed 
include the Structured Clinical Interview for DSM (SCID), the Diagnostic Interview Schedule 
(DIS), the Composite International Diagnostic Interview (CIDI), the World Health Organization 
(WHO) International Personality Disorders Examination (EPDE), and the WHO Schedules for 
Clinical Assessment in Neuropsychiatry (SCAN). These tests have been standardized around a 
core set of inquiries or directions. 

The response to standardized questions or directions is of interest for the assessor, both 
in terms of the substantive response as well as the behavior manifested in the delivery and of the 
response. Psychiatrists may want to measure behavioral responses, or observe the actions of 
psychiatrists in training. Cardiologists may want to measure physical responses to changes in 
applied stress, such as the EKG readings during a cardiac stress test. Lawyers may seek to 
identify inconsistencies between witnesses to an accident using standardized questions. Similar 
examples are found in virtually every profession where a pattern of testing for response and , 
analysis of the response is used. 

However, anyone who has participated in reviewing these interview or testing records 
realizes that the review process needs help and improvement. If there was .an efficient way to 
index the video, audio or other type of recording when the interview was recorded, much time 
could be saved when later searching the records for particular information. 

It is, therefore, an object of the present invention to provide a system and method for 
conducting, recording, playing back and editing or annotating digital recordings in audio, video, 
or other format. 

Key to any useful recording system based on digital media is the ability to efficiently 
generate meaningful indexes for the recorded material. Such indexes are well suited for 
structured information gathering events that use a systematic question-and-answer method 
containing a set or list of standardized inquiries, e.g., a list of television interview questions, a 
health questionnaire, a clinical or research medical assessment, a set of deposition questions, a 
list of actions to activate brain function in different ways, and a graded cardiac stress test, 
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among others. 

In the field of medicine and health, the established means of recording structured 
assessments has generally been by use of audio or video tape recorders. Indexing, if used at all, 
has been through length counters and manually added entries. This made the process tedious 
and often incomplete. In addition, the use of tape recorders means that the process of searching 
and playback is slow and inefficient. 

In the field of digital recording, many indexing schemes have been developed, ranging 
from low-level encoding to linking of text with video frames. Industry standard digital encoding 
methods such as the MPEG-2 (Society of Motion Picture Engineers) format, include channels 
for indexing based upon time code. However, most of these approaches generate index entries 
in a form not direcdy useful for playback and management, but rather provide basic 
mechanisms for supporting indexing. 

One system is known that uses computer voice-to-text processing to automatically 
generate words that are then used to index the digital audio recording. Although useful, it does 
not provide a more organized level of indexing, since many words will be used in many areas of 
a recording, and hence as such do not provide specific indexing to unique areas or segments of a 
recording. For indexing to be more useful, the index items should contain meaningful and 
unique information that is associated with the recording location or segment. Any indexing 
method can support this, but it is tedious and inefficient to generate such index items manually. 

Existing systems support some forms of indexing of recorded signals in a variety of 
ways. European patent application publication EP 0 495 612 by Lamming discloses a computer- 
based note-taking system integrated with an audio or video recording system. The computer 
presents a document editor style user interface to the user who either creates a new document or 
retrieves an existing document to which the user adds notes as a recording is made or played via 
the integrated audio or video system. As the user enters each note (mark or indicum), the 
indicum is added to the document and it is time stamped and stored in an information-to-time- 
stamp index. The time stamps are not visible to the user; they are stored with the computer's 
internal representation of the indica entered by the user. A video-frame time stamp function 
time stamps time code data received from the audio or video recorder and creates a time stamp- 
to-time-code index. A browser function permits the user to retrieve sections of the recording 
using the indicia directly by selecting the indicia. The browser looks up the indicia in the first 
index to retrieve the time stamp, and looks up the time code of the recording in the second index 
using the time stamp, playing the section of the recording in the area indicated by the time code. 
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EP 0495 612 also discloses how time stamping the indicia may be applied to creating topic or 
key word data. By entering new, separate indicia spatially near a previously entered indicia that 
is a key word or topic whenever an idea or speaker or topic applies to the previously entered 
indicia, later selection of all of the marks spatially associated with a topic will result in all 
sections of the recording indexed by the time stamps of the respective indicia to be replayed. 

U.S. Pat. No 4,841 ,387, entided "Arrangement for Recording and Indexing 
Information*' and issued to Rindfuss, discloses a system for recording information relating to an 
event on a recording medium, such as an audio or video tape, and for indexing positions of 
handwritten notations made on a touch sensitive device and concerning the event to positions on 
the recording medium in order to allow the user to identify portions of the handwritten notations 
for which review of the correlated material on the recorded medium is desired. In the recording 
mode, the device makes an audio recording of the event on a standard cassette tape. 
Simultaneously, the electronic touchpad senses the position of the user's handwritten notes on 
the writing surface, and provides this information to the microprocessor which correlates the 
record of the positions of the handwritten notations on each page with the position of the 
recorded information on the audio tape at corresponding instants in time. Real-time constrained 
correlation vectors representing rectangular areas of the display surface each containing a 
cohesive group of handwritten text are each combined with a tape position that correlates to the 
instant in time the handwriting within that area began. 

U.S. Pat. No. 4,425,586 issued to Miller discloses a system that combines a video tape 
recorder with a computer in such a manner that these two components each automatically 
record and display where related information is stored in its own mechanism as well as in its 
counterpart mechanism, allowing the user to determine the location of all the corresponding 
data stored both on video tape and on a storage medium such as a diskette, by examining only 
one storage medium. Notes about the recorded event or document may be entered onto the 
diskette along with the automatic entry of the corresponding reel number and frame number of 
the video record and diskette and file address number of the computer storage medium. Another 
feature disclosed is the capability of the system to enter and display the time and date on both 
the video tape and diskette recording mediums as well as on both video monitors along with the 
data address location information. 

U.S. Pat No. 4,924^87 issued to Jeppeson discloses a computerized court reporting 
system which provides for periodically annotating the stroke record made by the user of a court 
stenographic machine with a time stamp from a system clock while simultaneously sending a 
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time stamp to a recording system making a video and audio recording of the testimony. The 
logic of a control system determines automatically when to time stamp the stroke record and 
permits the user to trigger a control function to annotate the video recording with automatic "on 
the record" and "off the record" messages with associated time stamps. 

U.S. Pat. No. 5,564,005 issued to Weber et al. discloses a general system providing 
means for the user to first create a "time zone" correlated to a recording of an event, and then to 
flexibly enter into the time zone information relevant to the event, addressing some of the 
disadvantages of the systems described above. 

These real-time data correlation and access systems have similar disadvantages. In the 
case of U.S. Pat. No. 5,564,005, the system does not automatically generate meaningful indica 
at the time the "time zones" are established by the actions of the user. Rather, information about 
the events correlated to a time zone is entered later, in the form of handwritten strokes, or may 
be generated and entered in real-time, subject to the conceptual and processing speed limitations 
and variance of a user's abilities. Therefore, the primary initial index guide to the meaning of 
the events within a time zone are the temporal characteristics of the time zone, augmented at the 
time by the memory of the user. In all these systems, no method or user interface design is 
explicitly suggested for handling the case of repetition or extension of certain events or 
activities in the recording, using a common indica for events or activities thereby related but 
occurring at different times, thereby facilitating or enhancing a user's personal note-taking style 
or accommodating the note-taking function to a variety of applications. For all except U.S. Pat. 
No. 5,564,005, the user's ability to index notes to the recording is entirely controlled through 
the indicia, or notes, the user has entered in a document, since the time stamps or position 
captures are those made at the time the notes are entered. Those systems assume, therefore, that 
the time of entry of a note sufficiently corresponds with the time or positions of the recording to 
provide an adequate index into the recording, requiring the user to mentally process the ongoing 
real-time events and within a certain time variance generate and enter the note. In the case of 
U.S. Pat. No. 4,924,387, this provides an adequate indexing structure since the stroke record 
made is intended to be a verbatim transcription of the verbal testimony made in a courtroom. In 
the case of EP 0495 612, however, where a verbatim transcription of the event may not be the 
note-taker's intention, such an assumption does not allow for the later, real-time augmentation 
of previously entered notes, while still recording, with additional notes related to the previously 
entered notes, since the later notes will be time stamped with the time they were entered rather 
than with the time of the material to which they relate or are relevant. In the case of the system 
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disclosed by Rindfuss, notes may be entered later, but the later-entered notes will be correlated 
with the position of the tape at the time the later they relate or are relevant. In the case of the 
system disclosed by Miller, notes addressed to the diskette record of the event at a later time are 
entered entirely during a post-processing phase and not during the real-time recording of the 
event. In some of these systems, neither time stamps nor tape positions are visible to the user 
and so the temporal or spatial context of the entered indicia is not available to the user to 
enhance the retrieval function. In addition, in EP 0 495 612 the function provided for creating 
topic or key words from the entered indicia may be practically limited to one display "page" or 
screen unless the user reenters the topic or key word on a second screen or scrolls between 
screens to add a mark to a previously entered topic or key word As with augmentation of notes 
in general, there is no facility for associating a key word or topic name created at a later time 
with notes entered earlier. 

Existing post-processing (non-real-time) annotation systems in the field of post- 
production video editing provide for the creation of annotations about scenes correlated with 
"in" and "out" time codes identifying the scenes on a video recording. U.S. Pat No. 5,21 8,672 
is an example of such a system. It is disclosed there that scene descriptions may be revised after 
initial creation, but the correlation of the annotations are confined temporally to the identified 
scenes. 

In the post-processing system for the correlation of legal depositions with video 
recordings thereof disclosed in U.S. Pat. No. 5,172,281, a time code number is assigned by an 
operator of the system to both the computer transcript and the videotape segment where each 
question and answer passage begins, requiring manual time code entry. The location of 
individual words in the transcript may also be correlated with their corresponding position in 
the video recording, but there is no such result during the real-time process. In addition, the 
system does not appear to provide for the entry of notes or annotations. 

Despite the number of such prior art methods, there exists a need for a recorded 

information system that will digitally record events and automatically generate informative 
indica that are correlated in real-time with the recording of the events without affecting an 

interviewer's ability to take notes or assess the flow of events. It would be desirable to have a 

system for annotating an indexed digital recording of information in such a manner that the 

annotation did not disrupt the standardization of the index or otherwise adversely affect the 

ability of one set of recorded information in a standardized index key to be compared to another 

set of recorded information having the same standardized index key. 
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There is also a need to generate a standard set of indexes that are not subject to 
individual variance and which can be used between and among recorded information pertaining 
to different individuals or events so that comparisons can be made. In the case of video and 
audio logging tools, exclusively post-processing systems are inadequate for generating notes 
and relationships between recorded segments, and are time consuming because they require 
review of the entire tape in order to generate an index. Automatic indexing of video and audio 
notes by system time stamping of key- or handwritten strokes or automatic detection of speaker 
voice changes does not provide adequate context markers for the recorded signals or does not 
permit user control of the amount of detail to be captured. For example, an index created on the 
basis of speaker segmentation of the material would tell who was speaking but not the substance 
of the talk. Establishing time zones through the action of a user does not directly generate indica 
containing information regarding the events within the time zones. 



SUMMARY OF THE INVENTION 

It is an object of the invention to provide a system and method for digitally recording 
information about an event or individual and generating a standard set of indexes for this 
information. 

It is a further object of the invention to provide a system and method for annotating an 
indexed, digital recording of information without changing the standardized index markings in 
such a manner that would adversely affect the ability of one set of recorded information in a 
standardized index key to be compared to another set of recorded information having the same 
standardized index key. 

In accordance with a preferred embodiment of the present invention, the invention 
employs a system that includes: (a) means for presenting a subject or operator with a 
measurement item; (b) a recorder that records and stores observations of events associated with 
the measurement item in a digital recording that includes a data track sequenced by a time track 
of time data; and (c) indexing software that enters an index mark based on said time data into an 
editable database denoting said measurement item. Any annotation, cross referencing, or entry 
of notes will occur in the database in association with the time data track index without 
affecting the index. 

In use, the testing system of the invention includes the following steps; (a) presenting a 
measurement item to a subject or operator; (b) digitally recording on a digital storage medium at 
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least one event associated with said measurement item, said digital recording having a event 
data track of event data sequenced by a time track of time data; (c) indexing said digital 
recording by entering time data related to said event into a field on computer database along 
with an identifier of said measurement item; and (d) optionally, augmenting said computer 
database by adding information to said database that relates to said event field without changing 
the relationship of said time data to said event data. 

The system and method of the invention permit measurement items to be associated to 
observed events, the responses recorded and indexed by measurement items onto a computer- 
based database for review, annotation and augmentation of diverse information without 
affecting the time sequencing of the recorded data or the ability to replay a portion or all of the 
recorded events. Such a system will allow assessors and others using a "test and response" 
method of investigation much greater note-taking and annotation flexibility than was previously 
available with prior systems. The system of the invention also permits assessors to re-order 
recordings of events and compare similar events from different subjects- on the basis of the 
associated measurement items in order to investigate possible relationships and/or trends not 
previously discernible. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a flow diagram of the recording and indexing functions of the present 
invention. 

FIG. 2 is a flow diagram of the playback and editing functions of the present invention. 

DETAILED DESCRIPTION 

The invention is described with reference to inquiries from an assessor and responses 
made by an interviewee during an oral interview. It will be understood that the description is 
equally applicable to any form of assessment or observation by verbal, visual, audio, or physical 
method (e.g., questions or directions whether presented orally or in printed form, physical tests 
or testing, etc.) and any form of response by the subject, human or other, of observation (verbal, 
nonverbal, or physical response) with mere changes in semantics to describe the particular form 
of test and response. 
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A testing system according to the invention includes the following steps: (a) presenting 
a measurement item to a subject or operator; (b) making a digital record on a digital storage 
medium at least one event associated with said measurement item, said digital record having an 
event track of event data sequenced by a time track of time data; (c) indexing said digital 
recording by entering time data an identifier relating to the measurement item into a fields on a 
computer database; and (d) optionally, augmenting said computer database by adding formation 
to said database that relates to said event field without changing the relationship of said time 
data to said event data. 

An interviewing system according to the invention includes a personal computer having: 
(a) an output communication subsystem that allows communication with an interviewee or 
interviewer so that measurement items can be presented; (b) an input subsystem for the 
interviewee to respond to the measurement item communications; (c) a recording subsystem 
that makes a digital record of the output communications as well as the interviewee's responses 
with a data track of event data synchronized by a time track of time data; (d) a digital storage 
subsystem that allows the digital record to be stored for playback and editing; and (e) a 
computer with database software that can generate a database event index record based on said 
time data for each measurement item. 

The means for presenting the subject or operator with measurement items can take many 
forms depending on the type of measurement item involved with the assessment. The system 
and its method of use is most easily described in the context of an interview between a subject 
and an interviewer using a computer to display a series of questions to the interviewer as a 
guide for the interview. The interview is recorded by a recorder (preferably digital) that includes 
a data track synchronized by a time track of time data. As the interviewer moves from one 
question to another via input to the computer, the input is an act that causes the database to 
record the time data preferably a stop data bit for termination of the preceding segment and a 
start data bit for beginning the next segment) and an associated identifier that pertains to the 
new question (measurement item). 

As those skilled in the art will appreciate, the indexing act can be done in an active 
interview situation, by the interviewer, as questions are posed. The indexing can also be done in 
passive situation, where the operator is observing but not necessarily actively interacting, by the 
operator entering an input into the computer from a list of predefined topic items (measurement 
items) during actions by the subject of observation. Either method will generate a database of 
event data that is indexed by a series of identifiers that can be sorted and retrieved for selective 
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retrieval and playback of the events. 

Preferably, the software for a personal computer that has the capability to: (i) 
communicate inquiries to the interviewee directly or through an interviewer in a systematic 
manner, (ii) provide a rating scale or method by which said inquiries can be rated, and (iii) 
provide an index-in the form of retrievable, index marks associated with the inquiries. A 
personal computer is used to communicate the inquiries and then to capture and digitally 
record or control a recording device that records the interviewee's responses to form a 
digital record of indexed responses. 

During later playback and if responses are to be further indexed, the system can also 
insert additional response index marks and the identifiers for them into the digital record so that 
responses are digitally marked for retrieval. An assessor may also insert additional inquiry 
index marks into the recorded interview. Index marks can be placed at any point in the 
recording but an inquiry index mark is preferably inserted where a specified data item inquiry is 
being communicated, and a response index mark is inserted where the response is made. The 
inquiry and response index marks are desirably standardized and relate to the content of the 
inquiry or the response. 

The invention is conveniently explained in the context of an interview under the WHO 
Schedules for Clinical Assessment in Neuropsychiatry (SCAN). These schedules represent a 
detailed instrument for clinically-based, standardized psychiatric assessment of 
psychopathology. The SCAN is a widely recognized and used international standard. It will be 
understood by those skilled in the art that the techniques applied to a SCAN assessment would 
be applicable to other types of assessment instruments, physical examinations, or other physical 
phenomenon whose analysis is aided with a recording of sights, sounds, and/or other domains 
of observation with only minor semantic differences that describe the same essential types of 
events. 

During an assessment interview with a human subject, real-time selection of the relevant 
item in the natural course of the events automatically generates a series of time-stamped inquiry 
index marks that serve as an informative indicum for the recorded data. The time-stamp index 
marks time stamp plus identifier) are stored in database fields with time data that is correlated to 
a sequence of event data in the recording of the events. The resultant set of time stamp index 
marks then provides a general index structure which can be used to further support, 
independently of the real-time process, a flexible note-taking structure and user interface that 
complements diverse personal note-taking styles and application needs. 

10 
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Notations and annotations in the form of text, sound, video, or other data can be entered 
into the database as separate fields of information without affecting the relationship between the 
time stamp index mark and the recorded data associated therewith. Thus, the use of digital index 
marks of an event recording in an editable database provides assessors with the ability to 
generate assessment records rich in diverse content. When coupled with standardized 
assessment inquiries (e.g., . a list of standard interview questions) and a database field that 
identifies each inquiry by a standardized descriptor for each inquiry, this rich assessment record 
can be used as a basis for comprehensive comparisons on a rich, diverse content. 

In the case of assessment interviews in which there is no pre-existing set of 
measurement items (questions, definitions, inquiries, instructions, directions, or other tasks), an 
important feature of the present invention is the ability for the assessor or another individual (an 
interview designer) to generate a set of items prior to the event to be recorded and develop a 
new set of inquiries for investigating new issues. This set of items can be used and re-used as 
real-time index information designators which can later be used as standard access mechanisms 
into both the notes and the correlated recorded signals. These new inquiries, e.g., key words, 
labels, or any other pointer to information, can identify specific concepts in the user-produced 
information for which the user wants to have later access. An information designator is treated 
as an object that is easily created and reused and this novel implementation both in the user 
interface and data structure or the present invention provides a flexible and powerful indexing 
tool into the recorded signals in addition to the access provided by the recorded medium's 
address parameter. 

In an environment where a recording of events is being made simultaneously, the 
interviewer has limitations on what actions can be taken or additional items considered as the 
interview is in progress. Hie present invention produces a digital database of indexed inquiries 
and responses that can be used to organize the assessor's notes originally written in free text or 
graphical form but later entered into the database with minimal restriction as to the amount of 
that information or when it is entered. The inquiry and/or response index marks allow the 
assessor's notes to be correlated with the recorded signals of the event. Further, the present 
invention provides the assessor with the ability to modify any notes that were previously 
entered without regard to when the modifications are made or where they are located on the 
recording medium. 

The invention recognizes, moreover, that the temporal or spatial sequencing of 
information is itself often a significant clue to the content of the information and may aid a user 
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in recalling memories or in retrieving recorded information from a perceived event. The inquiry 
and response index marks of the present invention provide a mechanism for the user to control 
the playback sequence and thereby view the index with items sorted in, e.g., chronological 
order, type of inquiry (as denoted by the type of inquiry index encoded into the digital record), 
by type of response (also as denoted by the type of response index mark encoded into the digital 
record), as well as any other order desired by the assessor. 

The system of the present invention is particularly effective for video and audio 
interviews where the events include a time sequence of responses by the test subject. These 
events are recorded as a series of audio or video input data sequenced by a time track of time 
data. Because the amount of detail that may be captured for a particular time on an associated 
recorded medium is open-ended, a richer set of annotations may be made while the video is 
being digitally recorded in real time than is typically allowed in previous analog systems. The 
time zone and reusable information designator object features provide for treating the event as a 
whole by permitting later augmentation or modification of previously made notes. This relieves 
the note-taker of many of the pressures typically associated with taking interview notes in real 
time. 

It is another significant advantage that the system of the present invention operates in 
both real-time and post-processing environments. This feature recognizes that note-taking as a 
human activity is typically an interactive process. The output data structure (e.g., the video log), 
produced by the system of the present invention during a real-time note-taking session while the 
event is being recorded may be further modified and/or augmented by the same user or by 
different users during postprocessing review sessions occurring in parallel or seriatim. This 
facilitates and potentially reduces the time involved in the post-processing session. 

A system for correlating event data being recorded in real time to event information 
entered into the system by a system user includes input circuitry connected to a user input 
device for producing signals indicating actions of a system user; output circuitry connected to a 
display having area for presenting images; and a processor connected for receiving the signals 
from the input circuitry, and connected for providing images to the output circuitry. The system 
also includes a digital recording device in communication with a digital storage medium for 
producing a plurality of signals indicating recorded event data about an event for storage on the 
storage medium. The recording device is currently producing the plurality of signals and 
indicating the recorded event data and is connected for simultaneously providing address data 
items to the processor of the system and to the recording device. The recording device stores the 
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plurality of signals indicating the recorded event data on the storage medium so that a recording 
address value of each address data item provided by the address source indicates a portion of 
the recorded event data. The systems also includes memory for storing data including 
instruction data indicating instructions the processor executes. 

Stated another way, the system may be configured to operate in a post-processing 
environment so that an event that has already been recorded but not indexed is the event about 
which a log is constructed according to the present invention. The event that has already been 
recorded is digitally stored on a storage medium as a plurality of signals indicating recorded 
event data about the event. In this aspect of the invention, the system includes an output device 
for presenting the previously recorded signals thereon to the system user. This output device is 
in communication with the storage medium including the plurality of signals indicating 
recorded event data about the event. The output device is connected for providing a plurality of 
address data items to the processor of the system as user-requested address values; each address 
data item is a recording address value indicating a portion of the recorded event data. 
Preferably, the plurality of signals indicating the recorded event data includes the plurality of 
address data items, but the output device may itself generate them as the plurality of signals is 
being presented by the output device. 

The system of the present invention permits a system user to interactively produce 
stored information that is correlated with an "address". The "address" refers to a measurement 
dimension used to identify a portion of the stored information. When the stored information is 
being correlated with recorded signals, the address typically refers to the location of the 
recorded signals on the storage medium storing the recorded signals. However, the system of 
the present information is not limited to correlating information with recorded signals, and may 
be used in any situation where later access to the stored information would be facilitated by 
correlation with an address, and by having an access mechanism as provided by the 
"information designators". Generally, when the system user is producing a database of stored 
information, the address record will be time data taken from the time track of the recording of 
the events. 

The computer-controlled system of the present invention uses common, readily 
available hardware components. For an assessment interview delivered via computer, the 
system includes a computer with an image display that will display questions or directions, an 
input that receives signals from a user interaction device, e.g., keyboard, digital tablet, 
microphone, camera or other recording device that produces a data track synchronized by a time 
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track. The display can present the questions directly to the subject or, alternatively, can display 
a series of questions to an interviewer/assessor who then presents the questions to the subject. 
Some act taken to move from one question to another generates the database time entry or 
start/stop time entries as well as the index identifier that pertains to the question. 

Analog recording devices can be digitized or recorded directly as a digital recording that 
includes: (a) a data track of sound, image, or other event data that are synchronized by (b) a 
time track that includes time data correlated with event data. The use and existence of 
these two data tracks is typically transparent to the user of the equipment but can be read by 
a computer and used to generate database entries for each recorded event based on the time 
track data associated therewith from the digital recording. 

The user interaction devices useful in the present invention may include any one of a 
variety of input devices controllable by a human user. For example, the user interaction device 
may include a pointing and gesturing device to indicate actions, gestures, and strokes such as 
those produced when making handwriting gestures. A stylus or other form of handheld pen are 
examples of a suitable user interaction device. Other pointing devices such as a mouse or a 
trackball may also be suitable for indicating actions, gestures, and strokes. 

The present invention is further understood by reference to FIG. 1 which shows a flow 
diagram of the recording and indexing functions of the present invention. A digital recording 1 
of a series of events is made directly by a digital recorder or made into a digital formation 2 
from an analog recording. The indexing system 3 is used to generate a database 4 of index items 
5 based on measurement items 6. 

Suitable measurement items can vary over a wide range and will depend on the 
information sought from the test subject and the type of analysis intended. Examples of 
analysis include behavorial responses to audio and/or visual information (questions, directions, 
lights, sounds, etc.), physical examinations of laboratory specimens, oral depositions of 
witnesses, etc. In the case of a behavioral examination, measurement items 6 can be generated 
in process 10 from a standard assessment instrument 7, open ended questions 8 and any other 
inputs 9 desired by the assessor. Preferably, process 10 is a simple system that is readily 
repeated. Suitable examples include, a question outline or list. 

Similarly, FIG. 2 shows a flow diagram of the playback and editing functions of the 
present invention. Digital data 20 made of an event sequence track of event data is synchronized 
by an associated time track of time data is played back 21 for audio, video, and/or physiological 
events. Preferably, new index items 22 are created during a post-processing playback and 
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review phase but may also be created separately, for example as a pre-existing detailed set of 
measurement sub-items, or as additional notes during the original assessment process. Each 
index items is based on a link 23 between the time data in the time track with corresponding 
event sequence data and entered as index data 24 in a database. This database can include 
associated records that identify the index entries by a common set of labels, index attributes 25 
or other identifier that allow diverse event data from different subjects under a common 
framework. The resulting assessment forms a database file 26 that can be stored, shared, or 
integrated into other digital records as needed by the assessor. 

Although the foregoing disclosure relates to preferred embodiments of the invention, it 
is understood that these details have been given for the purposes of clarification only. Various 
changes and modifications of the invention will be apparent, to one having ordinary skill in the 
art, without departing from the spirit and scope of the invention. 
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